Affect and Emotions in Political Trust

Today’s Agenda

Audio Data in the Social Sciences and in Surveys
Emotion Analysis
Working Paper: Affect and Emotions in Political Trust

Audio Data in the Social Sciences and in Surveys

What is Audio Data?

Audio data = information represented in the form of sound waves captured in a digital format (e.g., WAV)

Figure 1: Waveform of an audio file (amplitude over time).

Sources for audio data for the social sciences: Interviews, social media (e.g. live streams, Youtube, TikTok), public speeches and debates, political talk shows and podcasts, press conferences, recorded survey answers.

Characteristics of Audio Data & Answers

Audio Data contains rich information including paralinguistic elements
- e.g., pitch, volume, laughter and sighs, tone of voice, pause and silence, emotional cues

“System 1” versus “System 2” language/answers (Lütters et al. 2018)
- System 1: intuitive, automatic, and fast thinking → spoken answers
- System 2: analytical, deliberate, and slow thinking → written answers

Audio Answers in Surveys

Spoken answers compared to written answers are longer, more elaborate and detailed (Gavras et al. 2022, Höhne & Gavras 2022, Lütters et al. 2018, Revilla et al. 2020)

Explanations:

Gavras et al. 2022: Audio formats facilitate answer process by enabling open narration, intuitive and spontaneous answers (≠ intentional and conscious text answers).
Revilla et al. 2020: Speaking requires less effort than typing and voice formats make survey answering easier/quicker (lower barriers!).

But increase response times and non-response rates (Revilla et al. 2020, Lütters et al. 2018)

Methods for the Analysis of Audio Data

Increasing number of methods to analyze audio data (interdisciplinary effort)

Examples of Methods:
- Natural Language Processing
- Automatic Speech Recognition (“speech-to-text”) (Landesvatter et al. 2023)
- Speaker Diarization / Speaker Identification
- Environmental Sound Analysis
- Speech Emotion Recognition

Emotion Analysis

Sentiment Analysis = Emotion Recognition?

Affect = “an umbrella term that is used to refer to both emotions and moods” (Lee, Dirks, and Campagna 2023, 549)
Sentiment: the valence of a feeling (e.g., positive versus negative)

The concept of emotions is not always clearly distinguished from similar phenomena such as mood, affect, and feeling. (Gabriel et al. 2023, 39)

Emotions: more complex and multi-dimensional state of feeling characterized by their intensity driven by different cognitive evaluations (e.g., anger towards politicians)

Speech Emotion Recognition

Why use speech (audio data) for Emotion Analysis?
- → Recognizing emotions from text is difficult because it is stripped of all paralinguistic and acoustic features

Speech Emotion Recognition (SER) is part of the field of Automated Emotion Recognition (intersects with Neurocomputing, Affective Computing)

Evolution from traditional Machine Learning models (e.g., logistic regression) to methods based on deep learning/neural networks, e.g. SpeechBrain (Ravanelli et al. 2021)

Paper: Affect and Emotions in Political Trust

Research Question and Motivation (1)

Research question: Do sentiment & emotions impact political trust judgments?

Substantive debate: Rational reasoning vs. affective states
- Lahno (2020): Game-theory & rational choice
- “affect-based” form of political trust (e.g., Theiss-Morse and Barton 2017)

Research Question and Motivation (2)

Today’s relevance of emotions
- Changing media environment (e.g., TikTok) → emotions (fear, anger/rage) used to influence people and their political trust
- Affective polarization

Methodological:
- Extend regression-based analysis through open-ended probing data
- Explore audio (and text) data to measure emotions & their influence on political trust

Theory and hypotheses

“Automatic Hot Cognition” (Lodge and Taber 2013): all sociopolitical concepts that have been evaluated in the past are affectively charged
- → affective charge is automatically activated within milliseconds on mere exposure to the concept
Open-ended survey answers on reasons for respondents’ trust include both non-neutral sentiment (H1a) as well as emotional, i.e., non-neutral language (H1b)
The sentiment or emotion expressed in open-ended survey answers is correlated with the closed-ended trust score
- Positive (negative) sentiment has a positive (negative) effect on political trust (H2a)
- Emotion of happiness (anger/sadness) has a positive (negative) effect on political trust (H2b)

Questionnaire Design

Political Trust Question	“How often can you trust the federal government in Washington to do what is right?” (ANES)	5 closed-ended categories [Never (0); Some of the time (1); About half of the time (2); Most of the time (3); Always (4)]
Probing Question	“The previous question was: ‘How often can you trust the federal government in Washington to do what is right?’. Your answer was: ‘About half of the time’. In your own words, please explain why you selected this answer.”	open-ended, audio request, SVoice tool (Höhne, Gavras and Qureshi 2021)

Data: Overview

Self-administered web survey (prolific), September 6 to October 27, 2023
U.S. non-probability sample; \(n\)=1,474 with 491 open-ended audio answers (33% response rate)
quota-based (U.S. Census Bureau 2015) with challenges in obtaining sufficient participants in the oldest age category (58+)

Methods

Data: Example

Political trust (0-4)	Probing question	Emotion (Speechbrain)	Sentiment (BERT)	Sentiment (GPT)
0	Government exists only to perpetuate itself. It does so by promulgating rules and raising taxes in order to keep control of the populace.	Anger	negative	negative
3	Because we are the strongest in the world. And we have ever done things that people's America has like.	Happiness	positive	positive
3	Again, Congress does what's best for the public. It's only fringe issues that make the news. Most of the stuff goes to them. Most people don't pay attention to it.	Neutral	neutral	positive
1	I don't have too much trust in the government these days. I think there's too much money involved in influencing politicians and laws. So I should have said never, but sometimes maybe.	Sadness	negative	negative

Results: Sentiment distribution (H1a, text data)

The commonly used trust in government question used in many surveys produces predominantly negative associations.

Figure 2: Sentiment Classification with three categories by classifier (BERT vs. GPT).
*Note.* n=491 open-ended answers.

Results: Sentiment & political trust (H2a, text data)

Associations with a positive sentiment positively influence trust scores, and vice versa
- → Associations and their sentiment matter

Figure 3: Linear model of sentiment and a five-category trust score (bi- and multivariate).
*Note.* GPT-based classification. Reference category is negative sentimemt.

Results: Emotions distribution (H1b, audio data)

Only 17% of the probing answers contain emotional speech.

Figure 4: Emotion Classification obtained from SpeechBrain.
*Note.* Analysis of n=491 open-ended answers. Number of observations for each sentiment category: 408 (neutral), 44 (angry), 18 (sad), 21 (happy).

Results: Emotions & political trust (H2b, audio data)

Respondents with a happy emotional condition have increased trust scores.

Figure 5: Linear model of emotion and a five-category trust score (bi- and multivariate).
*Note.* SpeechBrain-based classification.

Conclusion and discussion

Sentiment related to trust judgments, but no consistent effect of emotions (only happiness)
NLP & AI & New forms of data introduce new opportunities for measurement

Limitations: Non-probability sample; other probing wording; small response rates (33%); discrete emotions; combine text & audio/sound;

Survey setting not “emotional” enough? What is the setting?

How does SpeechBrain classify? Can we generate a human benchmark?

Other challenges: Privacy & reproducibility

Thank you for your Attention!

How could future research on voice and emotions look like?

Outlook on future research agenda

Audio data plays a crucial role for studying societies, as spoken language is one of humanity’s most important means of communication, expression and information exchange in various fields (e.g., public speeches and debates, political talk shows and podcasts, press conferences).
→ We need more applied research to gain more experiences with this type of data.

Subfields to research: speech-to-text algorithms (Landesvatter, Behnert, Bauer 2023, Meitinger, Sluis, Schonlau 2024), ethical considerations and privacy concerns, survey and experiments design to collect audio data, etc.

Multi-disciplinary approaches are useful (e.g., experts from linguistics, sociology, psychology, and computer science).

Speech-to-Text algorithms

Landesvatter, Camille, Jan Behnert, and Paul C. Bauer. 2023. “Comparing Speech-to-text Algorithms for Transcribing Voice Data from Surveys.” SocArXiv. October 10. doi:10.31235/osf.io/vk6wj.

Speechbrain

SpeechBrain: open-source deep learning toolkit built on PyTorch for speech and audio processing tasks
Emotion detection: 2021 pre-trained model based on wav2vec 2.0 architecture (orginally used for speech recognition)
- Used Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset for training (80% accuracy across different emotions)
Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset: ~12 hours of annotated recordings, encompassing dialogues from 10 speakers (utterances)
- six human evaluators assessed emotional categories of the database (three per utterance) (Busso et al. 2008)
- one of 3 most widely used SER datasets (Wang et al. 2021).

Speechbrain accuracy

Our validation ~ 40 answers (single rater)
- Seems to work but also difficult to identify emotions
Is the human ear reliable enough to recognize emotions (pitch, speed, pauses, etc.) & independently from the text?
- Requires training…

Sentiment accuracy

References

Gabriel, Maier, Masch, and Renner. 2023. Political Leaders, the Display of Emotions, and the Public: An Empirical Study on the Coverage and Effects of Emotions in German Politics. Nomos.

Gavras, Höhne, Blom, and Schoen. 2022. “Innovating the collection of open-ended answers: The linguistic and content characteristics of written and oral answers to political attitude questions.” Journal of the Royal Statistical Society. Series A, 185(3):872-890.

Grimmelikhuijsen, Stephan. 2012. “Linking Transparency, Knowledge and Citizen Trust in Government: An Experiment.” International Review of Administrative Sciences 78(1):50–73.

Höhne and Gavras. 2022. “Typing or Speaking? Comparing Text and Voice Answers to Open Questions on Sensitive Topics in Smartphone Surveys.” Available at SSRN: https://ssrn.com/abstract=4239015 or http://dx.doi.org/10.2139/ssrn.4239015.

Lee, Kurt, and Rachel L. Campagna. 2023. “At the Heart of Trust: Understanding the Integral Relationship Between Emotion and Trust.” Group & Organization Management 48(2):546–80.

Lodge, Milton, and Charles S. Taber. 2013. The Rationalizing Voter. Cambridge University Press.

Lütters, Friedrich-Freksa, and Egger. 2018.“Effects of Speech Assistance in Online Questionnaires.” Presented at the General Online Research Conference, Vol. 18.

Ravanelli, Parcollet, Plantinga, et al. 2021. “SpeechBrain: A General-Purpose Speech Toolkit.” arXiv.

Revilla, Couper, Bosch, and Asensio. 2020. “Testing the Use of Voice Input in a Smartphone Web Survey.” Social Science Computer Review 38(2):207–24.

Theiss-Morse and Dona-Gene Barton. 2017. “Emotion, Cognition, and Political Trust.” Pp. 160–75 in Handbook on Political Trust. Edward Elgar Publishing.